Explainability-aided Domain Generalization for Image Classification

Author: Supervisor: Robin M. Schmidt Dr. Massimiliano Mancini

Co-Supervisor:

Prof. Dr. Zeynep AKATA

Reviewer:

Prof. Dr. Philipp Hennig EBERHARD KARLS UNIVERSITAT TUBINGEN

A thesis submitted in fulfillment of the requirements

for the degree of Master of Science (M.Sc.) in Computer Science

in the

Department of Computer Science

Explainable Machine Learning Group

Abstract of thesis entitled

Explainability-aided Domain Generalization for Image Classification

Submitted by Robin M. Schmidt for the degree of Master of Science (M.Sc.) at the University of Tübingen in April, 2021

Traditionally, for most machine learning settings, gaining some degree of explainability that tries to give users more insights into how and why the network arrives at its predictions, restricts the underlying model and hinders performance to a certain degree. For example, decision trees are thought of as being more explainable than deep neural networks but they lack performance on visual tasks. In this work, we empirically demonstrate that applying methods and architectures from the explainability literature can, in fact, achieve state-of-the-art performance for the challenging task of domain generalization while offering a framework for more insights into the prediction and training process. For that, we develop a set of novel algorithms including DIvCAM, an approach where the network receives guidance during training via gradient based class activation maps to focus on a diverse set of discriminative features, as well as ProDROP and D-TRANSFORMERS which apply prototypical networks to the domain generalization task, either with self-challenging or attention alignment. Since these methods offer competitive performance on top of explainability, we argue that the proposed methods can be used as a tool to improve the robustness of deep neural network architectures. Copyright ©2021, by Robin M. Schmidt ALL RIGHTS RESERVED.

Declaration

I, Robin M. SCHMIDT, declare that this thesis titled "Explainability-aided Domain Generalization for Image Classification" which is submitted in fulfillment of the requirements for the degree of Master of Science in Computer Science represents my own work except where acknowledgements have been made. I further declare that this work has not been previously included as a whole or in parts in a thesis, dissertation, or report submitted to this university or to any other institution for a degree, diploma or other qualifications.

Contents

1 Introduction 1

2 Domain Generalization 3

2.1 Problem formulation 3

2.2 Related concepts and their differences 5

2.2.1 Generic Neural Network Regularization 6

2.2.2 Domain Adaptation 6

2.3 Previous Works 6

2.3.1 Learning invariant features 7

2.3.2 Model ensembling 8

2.3.3 Meta-learning 8

2.3.4 Data Augmentation 8

2.4 Common Datasets 9

2.4.1 Rotated MNIST 9

2.4.2 Colored MNIST 9

2.4.3 Office-Home 9

2.4.4 VLCS 11

2.4.5 PACS 11

2.4.6 Terra Incognita 11

2.4.7 DomainNet 11

2.4.8 ImageNet-C 11

2.5 Considerations regarding model validation 12

2.6 Deep-Dive into Representation Self-Challenging 12

3 Explainability in Deep Learning 15

3.1 Related topics 16

3.1.1 Model Debugging 16

3.1.2 Fairness and Bias 16

3.2 Previous Works 17

3.2.1 Visualization 17

Back-Propagation 17

Perturbation 18

3.2.2 Model distillation 19

Local approximations 19

Model Translation 19

3.2.3 Intrinsic methods 19

Attention mechanism 19

Text explanations 20

Explanation association 20

Prototypes 20

3.3 Explainability for Domain Generalization 22

4 Proposed Methods 23

4.1 Diversified Class Activation Maps (DIVCAM) 23

4.1.1 Global Average Pooling bias for small activation areas 25

4.1.2 Smoothing negative Class Activation Maps 25

4.1.3 Conditional Domain Adversarial Neural Networks 27

4.1.4 Maximum Mean Discrepancy 27

4.2 Prototype Networks for Domain Generalization 28

4.2.1 Ensemble Prototype Network 29

4.2.2 Diversified Prototypes (ProDrop) 30

4.2.3 Using Support Sets (D-TRANSFORMERS) 33

5 Experiments 35

5.1 Datasets and splits 35

5.2 Hyperparameter Distributions & Schedules 35

5.3 Results 35

5.4 Ablation Studies 37

5.4.1 Hyperparameter Distributions & Schedules 37

5.4.2 DIVCAM: Mask Batching 38

5.4.3 DIVCAM: Class Activation Maps 39

5.4.4 ProDrop: Self-Challenging 39

5.4.5 ProDrop: Intra-Loss 40

6 Conclusion and Outlook 43

Bibliography 45

A Domain-specific results 57

B Additional distance plots 63

List of Figures

2.1 Meta-distribution D generating source and unseen domain distributions 5

3.1 Class activation maps across different architectures 18

3.2 Prototypes for the MNIST and Car dataset 21

3.3 Image of a clay colored sparrow and its decomposition into prototypes 21

4.1 Visualization of the DIVCAM training process 24

4.2 Used class activation maps in DIVCAM-S throughout training 26

4.3 Domain-agnostic Prototype Network 28

4.4 Ensemble Prototype Network 29

4.5 Second data split pairwise prototype distances with $w_{c,j}=-1.0$ . 31

4.6 Second data split pairwise self-challenging prototype distances with $w_{c,j}=-1.0$ 33

B. 1 First data split pairwise prototype distances with $w_{c,j}=-1.0$ 63

B. 2 Third data split pairwise prototype distances with $w_{c,j}=-1.0$ 64

B. 3 First data split pairwise self-challenging prototype distances with $w_{c,j}=-1.0$ 64

B. 4 Third data split pairwise self-challenging prototype distances with $w_{c,j}=-1.0$

B. 5 First data split pairwise prototype distances with $w_{c,j}=0.0$ 65

B. 6 Second data split pairwise prototype distances with $w_{c,j}=0.0$

B. 7 Third data split pairwise prototype distances with $w_{c,j}=0.0$ 66

B. 8 First data split pairwise self-challenging prototype distances with $w_{c,j}=0.0$

B. 9 Second data split pairwise self-challenging prototype distances with $w_{c,j}=0.0$ 67

B. 10 Third data split pairwise self-challenging prototype distances with $w_{c,j}=0.0$ 68

List of Tables

2.1 Differences in learning setups 5

2.2 Samples for two different classes across domains for popular datasets 10

2.3 Reproduced results for Representation Self-Challenging using the official code base 14

5.1 Performance comparison of the proposed methods on the PACS dataset 36

5.2 Performance comparison across datasets 37

5.3 Performance comparison for official PACS splits outside of DomAINBED 38

5.4 Hyperparameters and distributions used for the mask batching ablation study 38

5.5 Hyperparameters and distributions used for the mask ablation study 39

5.6 Ablation study for the DIVCAM mask batching on the PACS dataset 40

5.7 Ablation study for the DIVCAM masks on the PACS dataset 41

5.8 Self-challenging performance comparison for different negative class weights 41

5.9 Self-challenging performance comparison for different intra factors 42

A. 1 Domain specific performance for the VLCS dataset 58

A. 2 Domain specific performance for the PACS dataset 59

A. 3 Domain specific performance for the Office-Home dataset 60

A. 4 Domain specific performance for the Terra Incognita dataset 61

A. 5 Domain specific performance for the DomainNet dataset 62

List of Algorithms

1 Spatial- and Channel-Wise RSC 14

2 Diversified Class Activation Maps (DIVCAM) 25

3 Prototype Dropping (ProDROP) 32

List of Abbreviations

ADAM ADAptive Moment Estimation

AUC Area Under the Curve

CAM Class Activation Maps

CDANN Conditional Domain Adversarial Neural Network

CE Cross-Entropy

CNN Convolutional Neural Network

CMNIST Colored MNIST

DA Domain Adaptation

DANN Domain Adversarial Neural Network

DeepLIFT Deep Learning Important Features

$\mathbf{DG}$ Domain Generalization

DivCAM Diversified Class Activation Maps

DNN Deep Neural Network

ERM Empirical Risk Minimization

$\mathbf{FC}$ Fully Connected Layer

GAN Generative Adversarial Network

$\mathbf{GAP}$ Global Average Pooling

Grad-CAM Gradient-weighted Class Activation Maps

HNC Homogeneous Negative Class Activation Maps

I.I.D. Independent and Identically Distributed

$\mathbf{KL}$ Kullback-Leibler

LIME Local Interpretable Model-agnostic Explanations

MAML Model-Agnostic Meta-Learning

MMD Maximum Mean Discrepancy

MTAE Multi-Task Autoencoder

NLP Natural Language Processing

ProDrop Prototype Dropping

$\mathbf{ReLU}$ Rectified Linear Unit

$\mathbf{RSC}$ Representation Self-Challenging

RKHS Reproducing Kernel Hilbert Space

RMNIST Rotated MNIST

SGD Stochastic Gradient Descent

SVM Support Vector Machine

TAP Threshold Average Pooling

$\mathbf{UML}$ Unbiased Metric Learning

List of Symbols

Chapter 2

${\mathbf{x}}_i$ instance of input features

${\mathbf{y}}_i$ corresponding label for input features

${\dot{\mathbf{y}}}_i$ instance of label one-hot encoding

$\left({\mathbf{x}}_i,{\mathbf{y}}_i\right)$ sample of input and label pair

$\hat{\mathbf{y}}$ predicted output label

Xrandom variable for input features

Yrandom variable for output labels

Cnumber of classes

Dtraining dataset

Dtraining distribution

Zfeature representation

$\tilde{\mathbf{z}}$ masked features

Xinput space

Youtput space

$z$ latent space

Θ parameter space

$\theta$ model parameters

$f_{\theta }$ model predictor

$\varphi$ feature extractor

$w$ classifier

Knumber of feature maps of the last convolutional layer

$R\left(f\theta \right)$ model risk

$R_{\mathrm{erm}}\left(f_{\theta }\right)$ empirical model risk

Lloss term

三 set of source environments

$\mathrm{\Phi }$ set of test environments

$\xi$ environment

① meta distribution

Uunlabeled dataset

Llabeled dataset

Hreproducing kernel Hilbert space

$\phi$ feature map induced by a kernel

${\mathrm{\Delta }}_{\xi }$ local envinronment bias

${\theta }_{\xi }$ environment parameters

${\mathbf{M}}_c$ class activation map for class $c$

${\mathrm{g}}_{\mathrm{z}}$ gradient with respect to the features

${\tilde{\mathrm{g}}}_{\mathrm{z}}$ average pooled gradient values

$H_{\mathbf{z}}$ feature map height

$W_{\mathbf{z}}$ feature map width

$y_c$ logit for class $c$

${\mathbf{m}}_{i,j}$ feature mask at spatial location(i,j)

Cchange vector after applying the mask

$q_p$ feature percentile threshold

$b_p$ batch percentile threshold

Chapter 3

$g_{{\theta }^{\prime }}$ interpretable model

$\mathcal{G}$ set of interpretable models

$\mathrm{\Pi }$ complexity measure

$\mathcal{P}$ set of prototypes

${\mathbf{p}}_{j}j$ -th prototype

$H_{\mathbf{p}}$ height of the prototypes

$W_{\mathbf{p}}$ width of the prototypes

$g_{{\mathbf{p}}_j}$ prototype unit

$g_{\mathbf{p}}$ prototype layer

$\bar{\mathbf{z}}$

${\mathrm{\Psi }}_j$ similarity map between $j$ -th prototype and latent representation

$ϵ$ numerical stability factor

$w_{c,j}$ classifier weight connecting the $j$ -th prototype unit and class $c$ logit

${\theta }_{\varphi }$ parameters of the feature extractor

${\theta }_w$ parameters of the classifier

${\ell }_2$ euclidean distance

$\lambda$ loss term weighting factor

Chapter 4

${\tau }_{tap}$ threshold for average pooling

$J_{>}^m$ Set of Top- $m$ negative classes

Uuniform probability matrix

$M_c^{\prime }$ probability map

$\omega$ domain predictor

d domain ground truth

$\eta$ ${\ell }^2$ regularization weighting factor

$k$ kernel function

Pset of source domain pairs -

$\varrho$ cosine distance

$S_c$ support set for class $c$

$\mathrm{\Gamma }$ key head -

$\mathrm{\Lambda }$ value head

Ω query head

$\mathrm{k}$ keys of the support set

q queries

Vsupport-set values -

Wquery image values

$\alpha$ dot similarity between keys and queries

$\tilde{\alpha }$ attention weights

Bbatch size

$\alpha$ learning rate

γ weight decay factor

Introduction

Modern machine learning solutions commonly rely on supervised deep neural networks as their default approach and one of the tasks that is commonly required and implemented in practice is image classification. In its simplest form, the used networks combine multiple layers of linear transformations coupled with non-linear activation functions to classify an input image based on its pixel values into a discrete set of classes. The chosen architecture, also known as the model, is then able to learn good parameters that represent how to combine the information extracted from the individual pixel values using additional labeled information. That is, for a set of images we know the correct class and can automatically guide the network towards well-working parameters by determining how wrong the current prediction is and in which direction we need to update the individual parameters, for our network to give a more accurate class prediction.

Obtaining this labeled information, however, is very tedious in practice and either requires a lot of manual human labeling or sufficient human quality assurance for any automatic labeling system. A commonly used approach to overcome this impediment is to combine multiple sources of data, that might have been collected in different settings but represent the same set of classes and have already been labeled a priori. Since the training distribution described by the obtained labeled data is then often different from the testing distribution imposed by the images we observe once we deploy our system, we commonly observe a distribution shift when testing and the network generally needs to do out-of-distribution predictions. Similar behavior can be observed when the model encounters irregular properties during testing such as weather or lighting conditions which have not been captured well by the training data. For many computer vision neural network models, this poses an interesting and challenging task which is known as out-of-distribution generalization where researchers try to improve the predictions under these alternating circumstances for more robust machine learning models.

In this work, we pick up on this challenge and try to improve the out-of-distribution generalization capabilities with models and techniques that have been proposed to make deep neural networks more explainable. For most machine learning settings, gaining a degree of explainability that tries to give humans more insights into how and why the network arrives at its predictions, restricts the underlying model and hinders performance to a certain degree. For example, decision trees are thought of as being more explainable than deep neural networks but lack performance on visual tasks. In this work, we investigate if these properties also hold for the out-of-distribution generalization task or if we can deploy explainability methods during the training procedure and gain, both, better performance as well as a framework that enables more explainability for the users. In particular, we develop a regularization technique based on class activation maps that visualize parts of an image that led to certain predictions (DIvCAM) as well as prototypical representations that serve as a number of class or attribute centroids which the network uses to make its predictions (ProDrop and D-TRANSFORMERS). Specifically, we deploy these methods for the domain generalization task where the model has access to images from multiple training domains, each imposing a different distribution, but without access to images from the immediate testing distribution.

From our experiments, we observe that especially DIvCAM offers state-of-the-art performance on some datasets while providing a framework that enables additional insights into the training and prediction procedure. For us, this is a property that is highly desirable, especially in safety-critical scenarios such as self-driving cars, any application in the medical field such as cancer or tumor prediction, or any other automation robot that needs to operate in a diverse set of environments. Hopefully, some of the methods presented in this work can find application in such scenarios and establish additional trust and confidence into the machine learning system to work reliable. All of our experiments have been conducted within the DOMAINBED domain generalization benchmarking framework and the respective code has been open-sourced. ${}^1$

Chapter 2 Domain Generalization

Machine learning systems often lack out-of-distribution generalization which causes models to heavily rely on the training distribution and as a result don't perform very well when presented with a different input distribution during testing. Examples are application scenarios where intelligent systems don't generalize well across health centers if the training data was only collected in a single hospital $[4$ , 31, 142] or when self-driving cars struggle under alternative lighting or weather conditions [36, 191]. Properties that are often interpreted falsely as part of the relevant feature set include backgrounds [17], textures [62], or racial biases [178]. Not only can a failure in capturing this domain shift lead to poor performance, but in safety-critical scenarios this can correspond to a large impact on people's lives. Due to the prevalence of this challenge for the wide-spread deployment of machine learning systems in diverse environments, many researchers tried to tackle this task with different approaches. In this chapter, we want to give a broad overview of the literature in domain generalization and prepare the fundamentals for the following chapters. If you are already familiar with the field, you can safely skip this chapter and only familiarize yourself with the used notation.

2.1 Problem formulation

Supervised Learning In supervised learning we are aiming to optimize the predictions $\hat{\mathbf{y}}$ for the values $\mathbf{y}\in \mathcal{Y}$ of a random variable $Y$ when presented with values $\mathbf{x}\in \mathcal{X}$ of a random variable $X$ . These predictions are generated with a model predictor $f_{\theta }(\cdot ):\mathcal{X}\to \mathcal{Y}$ that is parameterized by $\theta \in \mathrm{\Theta }$ , usually the weights of a neural network,and is assigning the predictions as $\hat{\mathbf{y}}=f_{\theta }(\cdot )$ . To improve our predictions,we utilize a training dataset containing $n$ input-output pairs denoted as $D={\left\{({\mathbf{x}}_i,{\mathbf{y}}_i)\right\}}_{i=1}^n$ where each sample $\left({\mathbf{x}}_i,{\mathbf{y}}_i\right)$ is ideally drawn identically and independently distributed (i.i.d.) from a single joint probability distribution $\mathcal{D}$ . By using a loss term $\mathcal{L}\left(\hat{\mathbf{y}},\mathbf{y}\right):\mathcal{Y}\times \mathcal{Y}\to {\mathbb{R}}^{+}$ ,which quantifies how different the prediction $\hat{\mathbf{y}}$ is from the ground truth $\mathbf{y}$ ,we would like to minimize the risk,

of our model. Since we only have access to the distribution $\mathcal{D}$ through a proxy in the form of the dataset $D$ ,we are instead using Empirical Risk Minimization (ERM):

by adding up the loss terms of each sample. One common choice for this loss term is the Cross-Entropy (CE) loss which is shown in Equation (2.3).

Here, ${\dot{\mathbf{y}}}_i$ is the one-hot vector representing the ground truth class, ${\hat{\mathbf{y}}}_i$ is the softmax output of the model, ${\hat{y}}_{i,c}$ and $y_{i,c}$ are the $c$ -th dimension of ${\hat{\mathbf{y}}}_i$ and ${\dot{\mathbf{y}}}_i$ respectively.

The occurring minimization problem is then often solved through iterative gradient-based optimization algorithms e.g. SGD [155] or ADAM [98] which perform on-par with recent methods for the non-convex, continuous loss surfaces produced by modern machine learning problems and architectures [163].

On top of that,the model predictor $f_{\theta }$ can be decomposed into two functions as $f_{\theta }=w\circ \varphi$ where $\varphi :\mathcal{X}\to \mathcal{Z}$ is an embedding into a feature space,hence sometimes called the feature extractor,and $w:\mathcal{Z}\to \mathcal{Y}$ which is called the classifier since it is a prediction from the feature space to the output space $\left[73,131\right]$ . This often allows for a more concise mathematical notation.

Domain Generalization The problem of Domain generalization (DG) builds on top of this framework,where now we have a set of training environments $\mathrm{\Xi }=\left\{{\xi }_1,\dots ,{\xi }_s\right\}$ ,also known as source domains,where each environment $\xi$ has an associated dataset $D_{\xi }={\left\{({\mathbf{x}}_i^{\xi },{\mathbf{y}}_i^{\xi })\right\}}_{i=1}^{n_{\xi }}$ containing $n_{\xi }$ i.i.d. samples from individual data distributions ${\mathcal{D}}_{\xi }$ . Note that,while related,the environments have different joint distributions,i.e. ${\mathcal{D}}_{{\xi }_i}\ne {\mathcal{D}}_{{\xi }_j}\mathrm{\forall }i\ne j$ . Here, ${\mathbf{x}}_i^{\xi }\in {\mathbb{R}}^m$ is the $i$ -th sample for environment $\xi$ representing an $m$ -dimensional feature vector (i.e. an image in our case) and ${\mathbf{y}}_i^{\xi }\in \mathcal{Y}$ is the corresponding ground truth class label over the $C$ possible classes. The one-hot vector representing the ground truth is denoted as ${\dot{\mathbf{y}}}_i^{\xi }$ . To clear up some notation,we sometimes omit $\xi$ where it is obvious. From these source domains, we try to learn generic feature representations agnostic to domain changes to improve model performance [166]. Simply, we try to do out-of-distribution generalization where our model aims to achieve good performance for an unseen test environment ${\xi }_t$ sampled from the set of unseen environments $\mathrm{\Phi }=\left\{{\xi }_1,\dots ,{\xi }_t\right\}$ with $\mathrm{\Xi }\cap \mathrm{\Phi }=\varnothing$ based on statistical invariances across the observed training (source) and testing (target) domains $\left[73,86\right]$ . For that,we try to minimize the expected target risk of our model as:

Since we don’t have access to ${\mathcal{D}}_{{\xi }_t}$ during training,one simple approach is to assume that minimizing the risk over all source domains in $\mathrm{\Xi }$ achieves good generalization to the target domain. That is,we disregard the separated environments:

Again, this can be written with the empirical risk as a simple sum over all the environments and their corresponding samples:

The difference of this approach when compared to ordinary supervised learning is shown on a high-level in Table 2.1. It may also be helpful to think about a meta-distribution $\mathfrak{D}$ (real-world) generating source environment distributions ${\mathcal{D}}_{\xi }^{\mathrm{\Xi }}$ and unseen testing domain distributions ${\mathcal{D}}_{\xi }^{\mathrm{\Phi }}$ as shown in Figure 2.1.

Figure 2.1: Meta-distribution $\mathfrak{D}$ generating source environment distributions (left) and unseen environment distributions (right), adapted from: [5]

Table 2.1: Differences in learning setups adapted from: [73]. For each environment $\xi$ the labeled and unlabeled datasets are denoted as $L_{\xi }$ and $U_{\xi }$ respectively.

Homogeneous and Heterogeneous Sometimes, domain generalization is also divided into homogeneous and heterogeneous subtasks. In homogeneous DG we assume that all domains share the same label space ${\mathcal{Y}}^{{\xi }_i}={\mathcal{Y}}^{{\xi }_j}={\mathcal{Y}}^{{\xi }_t},\mathrm{\forall }{\xi }_i\ne {\xi }_j\in \mathrm{\Xi }$ . On the contrary,the more challenging heterogeneous DG allows for different label spaces ${\mathcal{Y}}^{{\xi }_i}\ne {\mathcal{Y}}^{{\xi }_j}\ne {\mathcal{Y}}^{{\xi }_t},\mathrm{\forall }{\xi }_i\ne {\xi }_j\in \mathrm{\Xi }$ which can even be completely disjoint [110]. For this work, we assume a homogeneous setting.

Single- and Multi-Source There exist subtle differences within this task which are called single- or multi-source domain generalization. While multi-source domain generalization refers to the standard-setting we have just outlined, single-source domain generalization is a more generic formulation [221]. Instead of relying on multiple training domains to learn models which generalize better, single-source domain generalization aims at learning these representations with access to only one source distribution. Hence,our training domains are restricted to $\mathrm{\Xi }=\left\{{\xi }_1\right\}$ ,described by one dataset $D_{{\xi }_1}={\left\{({\mathbf{x}}_i^{{\xi }_1},{\mathbf{y}}_i^{{\xi }_1})\right\}}_{i=1}^{n_{{\xi }_1}}$ and modeling a single source distribution ${\mathcal{D}}_{{\xi }_1}$ . For example,this can be achieved by combining the different datasets or distributions similar to Equation (2.5) or mathematically ${\cup }_{\xi \in \mathrm{\Xi }}{D}_{\xi }$ . This is different from the ordinary supervised learning setup since we want to analyze the performance of the model under a clear domain-shift (i.e. out-of-distribution generalization). Keep in mind, that strong regularization methods will also perform well on this subtask. These cross-over and related techniques are described in the following section.

2.2 Related concepts and their differences

As already introduced, members of the causality community might know the task of domain generalization under the term learning from multiple environments $\left[9,73,143\right]$ and researchers coming from deep learning might know it under learning from multiple domains. While these two concepts refer to the same task, there exist quite a few related techniques which we want to highlight here and distinguish in their scope. In particular, we focus on "Generic neural network regularization" and "Domain Adaptation" since each of these are very closely related to domain generalization and sometimes hard to distinguish, if at all. The overview in Table 2.1, however, includes even more learning setups to properly position this concept into the machine learning landscape.

2.2.1 Generic Neural Network Regularization

In theory, generic model regularization which aims to prevent neural networks from overfitting on the source domain could also improve the domain generalization performance [86]. As such, methods like dropout [177], early stopping [30], or weight decay [136] can have a positive effect on this task when deployed properly. Apart from regular dropout, where we randomly disable neurons in the training phase to stop them from co-adapting too much, a few alternative methods exist. These include dropping random patches of input images (Cutout &HaS) [40, 172] or channels of the feature map (SpatialDropout) [185], dropping contiguous regions of the feature maps (DropBlock) [64], dropping features of high activations across feature maps and channels (MaxDrop) [139], or generalizing the traditional dropout of single units to entire layers during training (DropPath) [104]. There even exist methods like curriculum dropout [130] that deploy scheduling for the dropout probability and therefore softly increase the number of units to be suppressed layerwise during training.

Generally, deploying some of these methods when aiming for out-of-distribution generalization can be a good idea and should definitely be considered for the task of domain generalization.

2.2.2 Domain Adaptation

Domain Adaptation (DA) is often mentioned as a closely related task in domain generalization literature $\left[131,148,192\right]$ . When compared,domain adaptation has additional access to an unlabeled dataset from the target domain $\left[34,123\right]$ . Formally,aside from the set of source domains $\mathrm{\Xi }$ and the domain datasets $D_{\xi }$ ,as outlined in Section 2.1,we have access to target samples $U_{{\xi }_t}=\left\{{\mathbf{x}}_1^{{\xi }_t},\dots ,{\mathbf{x}}_{n_{{\xi }_t}}^{{\xi }_t}\right\}$ that are from the target domain ${\mathbf{x}}_i^{{\xi }_t}\sim {\mathcal{D}}_{{\xi }_t}$ but their labels remain unknown during training since we want to predict them during testing. As a result, domain generalization is considered to be the harder problem of the two. This difference is also shown in Table 2.1.

Earlier methods in this space deploy hand-crafted features to reduce the difference between the source and the target domains [126]. Like that, instance-based methods try to re-weight source samples according to target similarity $\left[68,85,202\right]$ ,or feature-based methods try to learn a common subspace $\left[14,54,69,119\right]$ . More recent works focus on deep domain adaptation based on deep architectures where domain invariant features are learned utilizing supervised neural networks [22, 29, 60, 67], autoencoders [209], or generative adversarial networks (GANs) [21, 169, 186]. These deep NN-based architectures significantly outperform the approaches for hand-crafted features [126].

Even though domain adaptation and domain generalization both try to reduce the dataset bias, they are not compatible with each other [65]. Hence, domain adaptation methods often cannot be directly used for domain generalization or vice versa [65]. For this work, we don't rely on the simplifying assumptions of domain adaptation, but instead, tackle the more challenging task of DG.

2.3 Previous Works

Most commonly, work in domain generalization can be divided into methods that try to learn invariant features, combine domain-specific models in a process called model ensembling, pursue meta-learning, or utilize data augmentation to generate new domains or more robust representations. Since literature in the domain generalization space is broad, we utilized Gulrajani and Lopez-Paz [73, Appendix A] for an overview and to identify relevant literature while individually adding additional works and more detailed information where necessary.

2.3.1 Learning invariant features

Methods that try to learn invariant features typically minimize the difference between source domains. They assume that with this approach the features will be domain-invariant and therefore will have good performance for unseen testing domains [86].

Some of the earliest works on learning invariant features were kernel methods applied by Muandet, Balduzzi, and Schölkopf [132] where they experimented with a feature transformation that minimizes the across-domain dissimilarity between transformed feature distributions while preserving the functional relationship between original features and targets. In recent years, there have been approaches following a similar kernel-based approach [115, 116], sometimes while maximizing class separability $\left[65,83\right]$ . As an early method,Fang,Xu,and Rockmore [52] introduce Unbiased Metric Learning (UML) with an SVM metric that enforces the neighborhood of samples to contain samples with the same class label but from other training domains.

After that, Ganin et al. [61] introduced Domain Adversarial Neural Networks (DANNs) using neural network architectures to learn domain-invariant feature representations by adding a gradient reversal layer. Recently, their approach got extended to support statistical dependence between domains and class labels [2] or considering one-versus-all adversaries to minimize pairwise divergences between source distributions [5]. Motiian et al. [131] use a siamese architecture to learn a feature transformation that tries to achieve semantical alignment of visual domains while maximally separating them. Other methods are also matching the feature covariance across source domains [150] or take a causal interpretation to match representations of features [122]. Huang et al. [86] have also shown that self-challenging (i.e. dropping features with high gradient values at each epoch) works very well.

Matsuura and Harada [127] use clustering techniques to split single-source domain generalization into different domains and then train a domain-invariant feature extractor via adversarial learning. Other works have also deployed similar approaches based on adversarial strategies [37, 92].

Li et al. [112] deploy adversarial autoencoders with maximum mean discrepancy (MMD) [72] to align the source distributions,i.e. for distributions ${\mathcal{D}}_{{\xi }_1},{\mathcal{D}}_{{\xi }_2}$ and a feature map $\phi :\mathcal{X}\to \mathcal{H}$ where $\mathcal{H}$ is a reproducing kernel Hilbert space (RKHS) this measure is defined as Equation (2.7).

Ilse et al. [88] extend the variational autoencoder [99] by introducing latent representations for environments ${\mathcal{Z}}_{\xi }$ ,classes ${\mathcal{Z}}_{\mathbf{y}}$ ,and residual variations ${\mathcal{Z}}_{\mathbf{x}}$ . Further,Li et al. [110] use episodic training i.e. they train a domain agnostic feature extractor $\varphi$ and classifier $w$ by mismatching them with an equivalent trained on a specific domain ${\varphi }_{\xi }$ and $w_{\xi }$ in combinations $\left({\varphi }_{{\xi }_1},w_{{\xi }_2},{\mathbf{x}}_i^{{\xi }_2}\right)$ and $\left({\varphi }_{{\xi }_2},w_{{\xi }_1},{\mathbf{x}}_i^{{\xi }_2}\right)$ and letting them predict data outside of the trained domain ${\xi }_1\ne {\xi }_2$ . Piratla,Netrapalli,and Sarawagi [145] also learn domain-specific and common components but the domain-specific parts are discarded after training. Li et al. [109] deploy a lifelong sequential learning strategy.

2.3.2 Model ensembling

Some methods try to associate model parameters with each of the training domains and combine them, often together with shared parameters, in a meaningful matter to improve generalization to the test domain. Commonly, the number of models in these type of architectures grow linearly with the number of source domains.

The first work to pose the problem of domain generalization and analyze it was Blanchard, Lee, and Scott [20]. In there,they use classifiers for each sample ${\mathbf{x}}^{\xi }$ denoted as $f_{\theta }\left({\mathbf{x}}^{\xi },{\mu }^{\xi }\right)$ where ${\mu }^{\xi }$ corresponds to a kernel mean embedding [133]. For theoretical analysis on such methods please see Deshmukh et al. [38] and Blanchard et al. [19]. Later on,Khosla et al. [96] combine global weights $\theta$ with local domain biases ${\mathrm{\Delta }}_{\xi }$ to learn one max-margin linear classifier (SVM) per domain as ${\theta }_{\xi }=\theta +{\mathrm{\Delta }}_{\xi }$ and finally combine them, which has recently been extended to neural network settings by adding an additional dimension describing the training domains to the parameter tensors [107]. Ghifary et al. [66] propose a Multi-task Autoencoder (MTAE) with shared parameters to the hidden state and domain-specific parameters for each of the training domains. Further, Mancini et al. [125] use domain-specific batch-normalization [89] layers and then linearly combine them using a softmax domain classifier. Other works utilize other domain-specific normalization techniques [166], linearly combine domain-specific predictors [124], or use more elaborate aggregation strategies [35]. Ding and Fu [42] use multiple domain-specific deep neural networks with a structured low-rank constraint and a domain-invariant deep neural network to generalize to the target domain. There have also been works that assign weights to mini-batches depending on their respective error to the training distributions [84, 159]. Jin et al. [94] use attention mechanisms to align the features of the different training domains.

2.3.3 Meta-learning

Meta-learning approaches provide algorithms that tackle the problem of learning to learn [161, 183]. As such, Finn, Abbeel, and Levine [56] propose a Model-Agnostic Meta-Learning (MAML) algorithm that can quickly learn new tasks with fine-tuning. Li et al. [108] adapt this algorithm for domain generalization (no fine-tuning) such that we can adapt to new domains by utilizing the meta-optimization objective which ensures that steps to improve training domain performance should also improve testing domain performance. Both approaches are not bound to a specific architecture and can therefore be deployed for a wide variety of learning tasks. These approaches recently got extended by two reg-ularizers that encourage general knowledge about inter-class relationships and domain-independent class-specific cohesion [46], to heterogeneous domain generalization [117], or via meta-learning a regularizer that encourages across-domain performance [15].

2.3.4 Data Augmentation

Data Augmentation remains a competitive method for generalizing to unseen domains [211]. Works in this segment try to extend the source environments to a wider range of domains by augmenting the available training environments. However, to deploy an efficient procedure for that, human experts need to consider the data at hand to develop a useful routine [73].

Several works have used the MIXUP [210] algorithm as a method to merge samples from different domains $\left[123,195,200,203\right]$ . Other works have also tried removing textural information from images [194] or shifting it more towards shapes [10, 135]. Carlucci et al. [28] used jigsaw puzzles of image patches as a classification task to show that this improves domain generalization while Volpi et al. [193] demonstrate that adversarial data augmentation on a single domain is sufficient. Further, Volpi and Murino [192] use popular image transformations (e.g. brightness, contrast, sharpness) with different intensity levels to train a more robust model, or Somavarapu, Ma, and Kira [174] use other stylizing techniques. Several methods also use GANs to augment the available training data $\left[151,167,219\right]$ or use other methods to generate synthetic domains [218]. Qiao, Zhao, and Peng [148] deploy an adversarial domain augmentation approach using a Wasserstein Auto-Encoder [184].

2.4 Common Datasets

There exist several datasets that are commonly used in domain generalization research. Here, we want to introduce the most popular choices as well as interesting datasets to consider. We give an overview over Rotated MNIST, Colored MNIST, Office-Home, VLCS, PACS, Terra Incognita, DomainNet, and ImageNet-C. Currently, the most popular choices include PACS, VLCS, and Office-Home.

2.4.1 Rotated MNIST

The Rotated MNIST (RMNIST) dataset [66] is a variation of the original MNIST dataset [105] where each digit got rotated by degrees $\{0,15,30,45,60,75\}$ . Each rotation angle represents one domain as shown in Table 2.2 for classes " 2" and " 4". The overall dataset in Gulrajani and Lopez-Paz [73] includes 70000 images from 10 homogeneous classes(0 - 9)each with dimension $1\times 28\times 28$ .

2.4.2 Colored MNIST

The Colored MNIST (CMNIST) dataset [9] is another variation of the original MNIST dataset [105]. The grayscale images of MNIST got colored in red and green. The respective label corresponds to a combination of digit and color where the color correlates to the class label with factors $\{0.1,0.2,0.9\}$ as domains and the digit has a constant correlation of 0.75 . Since this is a synthetic dataset, the factors can easily be adapted or extended to more domains. We report these numbers since they are used in Arjovsky et al. [9] and Gulrajani and Lopez-Paz [73]. Since the correlation factor between color and label varies between domains, this dataset is well-suited for determining a models capability of removing color as a predictive feature [9].

To construct the dataset Arjovsky et al. [9] first assign an initial label $\tilde{y}=0$ for digit $0-4$ and $\tilde{y}=1$ for digit $5-9$ . This initial label is then flipped with a probability of $25\mathrm{\%}$ to obtain the final label $y$ . Finally,we obtain the color $z$ by flipping the label $y$ with probabilities $p^d\in \{0.1,0.2,0.9\}$ depending on the domain. The image is then colored red for $z=1$ or green for $z=0$ [9]. Samples for both classes across domains can be seen in Table 2.2.

Overall, the dataset in Gulrajani and Lopez-Paz [73] contains 70000 images from 2 homogeneous classes (green $\mathrm{\&}$ red) of dimension $2\times 28\times 28$ .

2.4.3 Office-Home

The Office-Home dataset [190] provides 15588 images from 65 categories across 4 domains. The domains include Art, Clipart, Products (objects without a background), and Real-World (captured with a regular camera). Samples from these domains for the classes "alarm-clock" and "bed" can be seen in Table 2.2. On average, each class contains around 70 images with a maximum of 99 images in a category [190]. In Gulrajani and Lopez-Paz [73] they use dimension $3\times 224\times 224$ for each image.

Table 2.2: Samples for two different classes across domains for popular datasets

2.4.4 VLCS

The VLCS dataset [52] is a dataset that utilizes photographic datasets as individual domains. As such, it contains the domains PASCAL VOC (V) [51], LabelMe (L) [158], Caltech101 (C) [111], and SUN09 (S) [33]. In total, there are 10729 images from 5 classes. Samples for the classes "bird" and "car" can be seen in Table 2.2. In Gulrajani and Lopez-Paz [73] they use the dimension $3\times 224\times 224$ for each image.

2.4.5 PACS

The PACS dataset [107] consists of images from different domains including Photo (P), Art (A), Cartoon (C), and Sketch (S) as individual domains. As such, it extends the previously photo-dominated data sets in domain generalization [107]. It includes seven homogeneous classes (dog, elephant, giraffe, guitar, horse, house, person) across the four previously mentioned domains. Table 2.2 shows samples from the "dog" and "elephant" classes across all domains.

In total, PACS contains 9991 images which got obtained by intersecting classes from Caltech256 (Photo), Sketchy (Sketch) [160], TU-Berlin (Sketch) [49], and Google Images (all but Sketch) [107].

2.4.6 Terra Incognita

The Terra Incognita dataset is a subset of the initial Caltech camera traps dataset proposed by Beery, Horn, and Perona [17]. It contains photographs of wild animals taken by camera traps at different locations (IDs:38,43,46,100) which represent the domains. As such,the version used by Gulrajani and Lopez-Paz [73] contains 24788 images of 10 classes,each with size $3\times 224\times 224$ . Samples from two different classes can be seen in Table 2.2. The chosen locations represent the Top-4 locations with the largest number of images, each with more than 4000 images.

The main data challenges that arise in this dataset include illumination, motion blur, size of the region of interest, occlusion, camouflage, and perspective [17]. This includes animals not always being salient or them being small or far from the camera which results in only partial views of the animals' body being available [17].

2.4.7 DomainNet

The DomainNet dataset [141] contains six domains: clipart (48129 images), infographic (51605 images), painting (72266 images), quickdraw (172500 images), real (172947 images), and sketch (69128 images) for 345 classes. In total, it contains 586575 images that got accumulated by searching a category with a domain name in multiple image search engines and, as an exception, players of the game "Quick Draw!" for the quickdraw domain [141].

To secure the quality of the dataset, they hired 20 annotators for a total of 2500 hours to filter out falsely labeled images [141]. Each category has an average of 150 images for the domains clipart and infographic, 220 images for painting and sketch, and 510 for the real domain [141].

2.4.8 ImageNet-C

The ImageNet-C dataset [80] contains images out of ImageNet [157] permutated according to 15 corruption types each with 5 levels of severity which results in 75 domains. The types of corruptions are out of the categories "noise", "blur", "weather", and "digital" [80]. Table 2.2 shows a few of the available corruptions at severity 3 for the same image sample of two different classes. As their corruptions, they provide Gaussian Noise, Shot Noise, Impulse Noise, Defocus Blur, Frosted Glass Blur, Motion Blur, Zoom Blur, Snow, Frost, Fog, Brightness, Contrast, Elastic, and Pixelate [80]. Overall, the dataset provides all 1000 ImageNet classes where each image has the standard dimension of $3\times 224\times 224$ . Currently,this dataset is not implemented by Gulrajani and Lopez-Paz [73].

2.5 Considerations regarding model validation

In traditional supervised learning setups, we train our model on the training dataset, validate its hyperparameters (e.g. number of layers or hidden units in a neural network) on a separate validation dataset, and finally evaluate our model on an unused test dataset. Notice, that the validation dataset should be distributed identically to the test data to properly fit the hyperparameters of our architecture. This is not a straightforward process for domain generalization, as we lack a proper validation dataset with the needed statistical properties. There exist several approaches to this problem, some of them being more grounded than others. Here we give an overview of the three approaches outlined by Gulrajani and Lopez-Paz [73] that are respectively used in their DG framework DomAINBED.

Training-domain validation set In this approach, each training domain gets further split into training and validation subsets where all validation subsets across domains get pooled into one global validation set. We can then maximize the model's performance on that global validation set to set the hyperparameters. This approach assumes the similarity of training and test distributions.

Leave-one-domain-out cross-validation We can train $s$ models with equal hyperparameters based on the $s$ training domains where we each hold one of the domains out of training. This allows us to validate on the held-out domain and average among them to calculate the global held-out domain accuracy. Based on that, we can choose a model and re-train it on all of the training domains.

Test-domain validation set (oracle) A rather statistically biased way of validating the model's hyperparameters is incorporating the test dataset as a validation dataset. Because of this, it is considered bad style and should be avoided or at least explicitly marked. However, one method that is possible is to restrict the test dataset access as done by Gulrajani and Lopez-Paz [73] where they prohibit early stopping and only use the last checkpoint.

Other works have also come up with alternative methods to choose the hyperparameters. For example, Krueger et al. [101] validate the hyperparameters on all domains of the VLCS dataset and then apply the settings to PACS while D'Innocente and Caputo [35] use a validation technique that combines probabilities specific to their method.

2.6 Deep-Dive into Representation Self-Challenging

Since some of our proposed methods use ideas from Representation Self-Challenging (RSC) [86], we explain their approach more in detail here. They deploy two RSC variants called Spatial-Wise RSC and Channel-Wise RSC which they randomly alternate between. Generally, these are shown in Algorithm 1 and operate on features after the last convolutional layer.

First, RSC calculates the gradient of the upper layer with respect to the latent feature representation according to Equation (2.8). Here, $\odot$ is the element-wise product and $\dot{\mathbf{y}}$ is the one-hot encoding of

the ground truth.

Afterward,they average-pool the gradients to obtain ${\tilde{\mathbf{g}}}_{\mathbf{z}}$ . The key difference between the Spatial-Wise and Channel-Wise RSC lies in the average pooling done to compute ${\tilde{\mathbf{g}}}_{\mathbf{z}}$ in line 5 and the duplication in line 6 in Algorithm 1. While for Spatial-Wise RSC average pooling is done on the channel dimension according to Equation (2.9) yielding ${\tilde{\mathbf{g}}}_{\mathbf{z},i,j}\in {\mathbb{R}}^{H_{\mathbf{z}}\times W_{\mathbf{z}}\times 1}$ for spatial location(i,j),in Channel-Wise RSC the same computation is done on the spatial dimension with Equation (2.10) yielding ${\tilde{\mathbf{g}}}_{\mathbf{z}}\in {\mathbb{R}}^{1\times 1\times K}$ , a vector with the size of the feature map count.

Depending on which dimensions are missing to get back to the original size of $\mathbf{z},{\mathbf{g}}_{\mathbf{z}}\in {\mathbb{R}}^{H_{\mathbf{z}}\times W_{\mathbf{z}}\times K}$ ,the computed values get duplicated along these dimensions. In the case of the Spatial-Wise RSC these are the channels, while for the Channel-Wise RSC these are the spatial dimensions.

Next,Huang et al. [86] compute the(100 - p)th percentile with the threshold value as $q_p$ and compute the mask ${\mathbf{m}}_{i,j}$ for spatial location(i,j)based on Equation (2.11). This mask is set to 0 for the corresponding Top- $p$ percentage elements in ${\tilde{\mathbf{g}}}_{\mathbf{z}}$ and therefore has the same shape.

Huang et al. [86] apply the computed mask on the feature representation to yield ${\tilde{\mathbf{z}}}_p=\mathbf{z}\odot \mathbf{m}$ which they validate using Equation (2.12). This computes the difference with and without the masking in the correct class probabilities and yields a difference score for each sample in the vector $\mathbf{c}$ .

A positive value represents that the masking for that sample made the classifier less certain about the correct class while a negative value represents the opposite and made the classifier more certain about the correct class. Similar to previously,Huang et al. [86] calculate Top- $p$ of the positive values with the threshold as $b_p$ . They revert the whole masking for all Top- $p$ samples inside each batch according to Equation (2.13) where each spatial location(i,j)of the mask associated with sample $n$ gets set back to 1 if the condition applies, otherwise the mask values remain unchanged.

Finally,we mask the features with the obtained final mask to obtain $\tilde{\mathbf{z}}=\mathbf{z}\odot \mathbf{m}$ ,compute the loss $\mathcal{L}\left(w\left(\tilde{\mathbf{z}}\right),\mathbf{y}\right)$ and backpropagate to the whole network.

Problems Interestingly, since many architectures like ResNet-18/ResNet-50 deploy average pooling in their forward pass after the last convolutional layer, naïve Spatial-Wise RSC doesn't make sense since average pooling is done along the channel dimension and such architectures additionally average pool on the spatial dimension. This results in feature values getting spread evenly across the image regardless of the masking. Even though this isn't mentioned in their paper, they address this issue in their official repository and propose an alternative computation. For that, they calculate the mean

Algorithm 1: Spatial- and Channel-Wise RSC

Input: Data $\mathbf{X},\mathbf{Y}$ with ${\mathbf{x}}_i\in {\mathbb{R}}^{H\times W\times 3}$ ,drop factor $p$ ,epochs $T$

while epoch $\le T$ do

for every sample (or batch) \( \mathbf{x},\mathbf{y} \) do

对每个样本（或批次）\( \mathbf{x},\mathbf{y} \)执行

	Extract features \( \mathbf{z} = \phi \left( \mathbf{x}\right) \) // \( \mathbf{z} \) has shape \( {\mathbb{R}}^{{H}_{\mathbf{z}} \times  {W}_{\mathbf{z}} \times  K} \)

	提取特征\( \mathbf{z} = \phi \left( \mathbf{x}\right) \) // \( \mathbf{z} \)的形状为\( {\mathbb{R}}^{{H}_{\mathbf{z}} \times  {W}_{\mathbf{z}} \times  K} \)

	Compute gradient \( {\mathbf{g}}_{\mathbf{z}} \) w.r.t features according to Equation (2.8)

	根据公式(2.8)计算特征的梯度\( {\mathbf{g}}_{\mathbf{z}} \)

	Compute \( {\widetilde{\mathbf{g}}}_{\mathbf{z},i,j} \) by avg. pooling using \( {50}\% \) Equation (2.9) or \( {50}\% \) Equation (2.10)

	使用公式(2.9)或公式(2.10)通过平均池化计算\( {\widetilde{\mathbf{g}}}_{\mathbf{z},i,j} \)

	Duplicate \( {\widetilde{\mathbf{g}}}_{\mathbf{z}} \) along channel/spatial dimension for initial shape

	沿通道/空间维度复制\( {\widetilde{\mathbf{g}}}_{\mathbf{z}} \)以恢复初始形状

	Compute mask \( {\mathbf{m}}_{i,j} \) according to Equation (2.11)

	根据公式(2.11)计算掩码\( {\mathbf{m}}_{i,j} \)

	Mask features to obtain \( {\widetilde{\mathbf{z}}}_{p} = \mathbf{m} \odot  \mathbf{z}\; \) // Evaluate effect of preliminary mask \( \downarrow \)

	掩码特征以获得\( {\widetilde{\mathbf{z}}}_{p} = \mathbf{m} \odot  \mathbf{z}\; \) // 评估初步掩码\( \downarrow \)的效果

	Compute change \( \mathbf{c} \) according to Equation (2.12)

	根据公式(2.12)计算变化\( \mathbf{c} \)

	Revert masking for specific samples according Equation (2.13)

	根据公式(2.13)对特定样本还原掩码

	Mask features \( \widetilde{\mathbf{z}} = \mathbf{m} \odot  \mathbf{z} \)

	掩码特征\( \widetilde{\mathbf{z}} = \mathbf{m} \odot  \mathbf{z} \)

	Compute loss \( \mathcal{L}\left( {w\left( \widetilde{\mathbf{z}}\right) ,\mathbf{y}}\right) \) and backpropagate to whole network

	计算损失\( \mathcal{L}\left( {w\left( \widetilde{\mathbf{z}}\right) ,\mathbf{y}}\right) \)并对整个网络进行反向传播

end

结束

end

${\tilde{\mathbf{g}}}_{\mathbf{z},i,j}$ from Equation (2.9) on the gradients of features from the previous convolutional layer,instead of the last one, and downsample it by factor 0.5 to match the size.

Our Results In an effort to provide somewhat fair results which aren't too optimistic and neither too penalizing, we run the original RSC code five times for each of the testing environments and compute the average performance in Table 2.3.

Table 2.3: Reproduced results for Representation Self-Challenging using the official code base on the PACS dataset and with a ResNet-18 backbone.

Given these observations, we follow other works such as Nuriel, Benaim, and Wolf [137] and report our reproduced results whenever comparing to RSC.

Explainability in Deep Learning

Machine Learning systems and especially deep neural networks have the characteristic that they are often seen as "black-boxes" i.e. they are hard to interpret and pinpointing as a user how and why they converge to their prediction is often very difficult, if not impossible. Neural networks commonly lack transparency for human understanding [180]. This property becomes a prominent impediment for intelligent systems deployed in impactful sectors like, for example, employment [24, 149, 216], jurisdiction [74], healthcare [140], or banking loans where users would like to know the deciding factors for decisions. As such, we would like systems that are easily interpretable, relatable to the user, provide contextual information about the choice, and reflect the intermediate thinking of the user for a decision [199]. Since these properties are very broad, it is not surprising that researchers in this field have very different approaches. For this chapter, we used the field guide by Xie et al. [199] to paint an appropriate overview and properly introduce the different approaches. Commonly, methods try to provide better solutions with respect to Confidence, Trust, Safety, and Ethics to improve the overall explainability of the model [199]:

Confidence The confidence of a machine learning system is high when the "reasoning" behind a decision between the model and the user matches often. For example, saliency attention maps [87, 138] ensure that semantically relevant parts of an image get considered and therefore increase confidence.

Trust Trust is established when the decision of an intelligent system doesn't need to be validated anymore. Recently, many works have studied the problem of whether a model can safely be adopted $\left[63,93,188\right]$ . To be able to trust a model,we need to ensure satisfactory testing of the model and users need experience with it to ensure that the results commonly match the expectation [199].

Safety Safety needs to be high for machine learning systems that have an impact on people's lives in any form. As such, the model should perform consistently as expected, prevent choices that may hurt the user or society, have high reliability under all operating conditions, and provide feedback on how the operating conditions influence the behavior.

Ethics The ethics are defined differently depending on the moral principles of each user. In general, though, one can create an "ethics code" on which a system's decisions are based off [199]. Any sensitive characteristic e.g. religion, gender, disability, or sexual orientation are features that should be handled with great care. Similarly, we try to reduce the effect of any features that serve as a proxy for any type of discrimination process e.g. living in a specific part of a city, such as New York City's Chinatown, can be a proxy for the ethical background or income. Since this chapter gives a high-level overview of recent advances in explainability for neural networks, also concerning domain generalization, it is up to the reader's background if this is necessary.

3.1 Related topics

There exist several concepts which are related to explainable deep learning. Here, we explicitly cover model debugging which tries to identify aspects that hinder training inference, and fairness and bias which especially tackles the ethics trait to search for differences in regular and irregular activation patterns to promote robust and trustworthy systems [199].

3.1.1 Model Debugging

Model debugging, similar to traditional software debugging, tries to pinpoint aspects of the architecture, data-processing, or training process which cause errors [199]. It aims at giving more insights into the model, allowing easier solving of faulty behavior. While such approaches help to open the black-box of neural network architectures, we handle them distinctly from the other literature here.

Amershi et al. [7] propose MODELTRACKER which is a debugging framework and interactive visualization that displays traditional statistics like Area Under the Curve (AUC) or confusion matrices. It also shows how close samples are in the feature space and allows users to expand the visualization to show the raw data or annotate them. Alain and Bengio [3] deploy linear classifiers to predict the information content in intermediate layers where the features of every layer serve as input to a separate classifier. They show that using features from deeper layers improves prediction accuracy and that level of linear separability increases monotonically. Fuchs et al. [59] introduce neural stethoscopes as a framework for analyzing factors of influence and interactively promoting and suppressing information. They extend the ordinary DNN architecture via a parallel two-layer perceptron at different locations where the input are the features from any layer from the main architecture. This stethoscope is then trained on a supplemental task and the loss is back-propagated to the main model with weighting factor $\lambda$ [59]. This factor controls if the stethoscope functions analytical $\left(\lambda =0\right)$ ,auxiliary $\left(\lambda >0\right)$ , or adversarial $\left(\lambda <0\right)$ [59]. Further,Kang et al. [95] use model assertions which are functions for a model's input and output that indicate when errors may be occurring. They show that with these they can solve problems where car detection in successive frames disappears and reappears [95]. Their model debugging is therefore implemented through a verification system [199].

3.1.2 Fairness and Bias

To secure model fairness, there exist several definitions which have emerged in the literature in recent years. Group fairness [25], also known as demographic parity or statistical parity, aims at equalizing benefits across groups with respect to protected characteristics (e.g. religion, gender, etc.). By definition,if group $A$ has twice as many members as group $B$ ,twice as many people in group $A$ should receive the benefit when compared to $B$ [199]. On the other hand,individual fairness [47] tries to secure that similar feature inputs get treated similarly. There also exist other notions of fairness such as equal opportunity [75], disparate mistreatment [204], or other variations [78, 197].

Methods that try to ensure fairness in machine learning systems can be classified into three approaches which operate during different steps called pre-processing, in-processing, post-processing:

1. Pre-processing methods adapt the input data beforehand to remove features correlated to protected characteristics. As such, they try to learn an alternative feature representation without relying on these types of attributes [70, 120, 144, 208].

2. In-processing approaches add adjustments for fairness during the model learning process. This way,they punish decisions that are not aligned with certain fairness constraints $\left[1,45,48\right]$ .

3. Post-processing methods adjust the model predictions after training to account for fairness. It is the reassignment of class labels after classification to minimize classification errors subject to a particular fairness constraint [53, 75, 146].

3.2 Previous Works

Generally, we can divide methods for explainable deep neural networks in visualization, model distillation, and intrinsic methods [199]. While visualization methods try to highlight features that strongly correlate with the output of the DNN, model distillation builds upon a jointly trained "white-box" model, following the input-output behavior of the original architecture and aiming to identify its decision rules [199]. Finally, intrinsic methods are networks designed to explain their output, hence they aim to optimize both, its performance and the respective explanations [199].

3.2.1 Visualization

Commonly, visualization methods use saliency maps to display the saliency values of the features i.e. to which degree the features influence the model's prediction [199]. We can further divide visualization methods into back-propagation and perturbation-based approaches where they respectively determine these values based on the volume of the gradient or between modified versions of the input [199].

Back-Propagation

These approaches stick to the gradient passed through the network to determine the relevant features. As a simplistic baseline, one can display the partial derivative with respect to each input feature multiplied by its value [199]. This way, Simonyan, Vedaldi, and Zisserman [170] and Springenberg et al. [175] assess the sensitivity of the model for input changes [199]. This can also be done for collections of intermediate layers $\left[11,129,168,206\right]$ .

Zhou et al. [217] introduce class activation maps (CAMs) which are shown in Figure 3.1 based on global average pooling (GAP) [118]. With GAP, they deploy the following CNN structure at the end of the network: GAP (Convs) $\to$ Fully Connected Layer (FC) $\to$ softmax where CAMs ${\mathbf{M}}_c$ for each class $c$ are then calculated according to Equation (3.1). Here, $K$ are the number of convolutional filters, $\mathbf{z}$ are the activations of the last convolutional layer,and $w_{k,c}$ indicate the weights from the feature map $k$ of the last Convolutional Layer to logit for class $c$ of the FC [217].

By upsampling the map to the image size, they can visualize the image regions responsible for a certain class [217]. Therefore, every class has its own class activation map. The drawback of their approach is,that their method can only be applied to networks that use the GAP (Convs) $\to$ Fully Connected Layer (FC) $\to$ softmax architecture at the end [199].

Figure 3.1: Class activation maps across different architectures: [217]

Selvaraju et al. [164] solve this impediment by generalizing CAMs to gradient-weighted class activation maps (Grad-CAMs). Since their approach only requires the final activation function to be differentiable,they are generally applicable to a broader range of CNN architectures [164, 199]. For that,they compute an importance score ${\tilde{\mathbf{g}}}_{\mathbf{z},c}^k$ as:

Here, $y_c$ is the score before softmax and we calculate the gradient with respect to the feature map ${\mathbf{z}}^k$ in the final convolutional layer for every neuron positioned at(i,j)in the $H_{\mathbf{z}}\times W_{\mathbf{z}}$ feature map $\left[164,199\right]$ . Afterward,these importance scores get linearly combined for every feature map as shown in Equation (3.3) where they get additionally passed through a ReLU function:

This computation inherently yields a $H_{\mathbf{z}}\times W_{\mathbf{z}}$ importance map ( $14\times 14$ for VGG [171] and AlexNet $\left[\begin{array}{l}100\end{array}\right],7\times 7$ for ResNet [77]) where we upsample it using bilinear interpolation onto the image size to yield the Grad-CAM.